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Lessons from Recent Web Surveys 
at Harvard University 



Abstract 



This paper provides an overview of the entire process necessary to 
developing a university- wide web survey, from the community-building 
process for creating support for the survey and determining the questions, to 
the specific tasks necessary for designing and administering an efficient web 
product. 
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Lessons from Recent Web Surveys at 
Harvard University 



Overview of Essential Steps for Web Survey Projects 

I. Survey Background 

1. Critical background questions 

2. Building a survey team 

3. Defining survey content 

4. Choosing web surveys over paper surveys 

5. Selection of the survey population 

II. Designing a Web Survey 

6. Questionnaire design 

7. Choosing HTML software and format 

8. Turning a survey questionnaire into a web survey: technical 
issues 

III. Administering a Web Survey 

9. Administering a web survey: technical issues 

10. Pre-testing 

11. Privacy issues 

12. Survey incentives 

13. Technical support for duration of survey 

14. Output of survey data 



IV. Reportage 
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I. Survey Background 

In the past few years, several Harvard surveys that had been previously done as paper 
surveys were transformed into web surveys. These were high-profile surveys in that they 
were endorsed by administrators such as the Provost and Dean of the College, and 
covered major portions of the university populations (all undergraduates, all graduate 
students, tenured and non-tenured faculty). As recently as 2001, when planning for these 
surveys had begun, web surveys were still avant-garde. Diliman’s 2000 reprinting of 
Mail and Internet Sur\’eys devoted only 29 out of 450 pages to web surveys, indicative of 
the small size of the collected body of knowledge about web surveys at that time. 

1.1. Critical Background Questions 

The beginnings of our projects were similar to the beginnings of all good survey 
projects. There were four predominant reasons for conducting the surveys: 



Table 1 

Reasons to Do a Survey 


Top-Down 


University administration wants to know more about a population or 
subject area. 


Bottom-Up 


University department wants to make a policy or business-practice 
changes, and needs data to support recommendation. 


Consortial 

Project 


University participates in consortial project to have peer information. 


Trend Data 


University repeats previous survey in order to have trend data for 
decision-making . 
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For each project, the critical background questions were the same: 

Table 2 

Critical Background Questions 

1 . What is the purpose of the survey, and how will we use the survey data? 

2. Whom are we surveying? 

3. Who are the audiences for the survey results? 

4. What kind of support do we need for the survey effort? 

■ Building a survey team (see below) 

5. What survey methodology will we use? (see below) 

6. What is the project budget? 



The projects that stemmed from the Harvard Office of Instructional Research and 
Evaluation were all established consortial or trend-data surveys which already received 
support from the Dean of the College and had long-standing audiences for the survey 
results. However, a decision was made to switch both types of surveys from paper 
surveys to web surveys. 

The two surveys initiated by Harvard Real Estate Services (HRES) had more complex 
origins. HRES has traditionally conducted triennial surveys on housing and 
transportation topics. These surveys have focused on different University populations 
with the aim to collect updated demographic information, information on current housing 
decisions and satisfaction, and information about the demand and need for Harvard 
housing and transportation services. One of the surveys that we are discussing here, the 
2001 Harvard Graduate Student Housing Survey, had been previously done as a paper 
survey. There was a need for trend data, more complete coverage of the topic (and thus, 
a re-designed survey), and a desire for a more streamlined administration via the web. 

The Provost’s Advisory Committee on Housing then recognized a need to understand 
faculty need and demand for housing and housing services, and the Committee 
recommended that a survey be undertaken in the spring of 2003 for the first time. Again, 
a decision was made to conduct the 2003 Faculty Housing and Transportation Survey via 
the web. 
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1.2. Building a Survey Team 

For the two original-design HRES survey projects, the members of the HRES Core 
Survey Team were charged to do the following: 

Table 3 

Charge to Core Team Members 

1. Identify areas of expertise needed to undertake the effort (both functional and topic 

specific) 

2. Identify if there are survey synergies and multiple topics that can be combined to 
prevent over-surveying the population 

3. Identify strengths and gaps across the internal resources 

4. Determine if budget allows for external professionals 



To supplement HRES’s own expertise, several different types of consultants were 
retained to help with each survey (see Table 4 below). In addition, since the two HRES 
surveys were expected to have many and varied audiences for the results, a decision was 
made to establish different internal survey teams and advisory groups to secure buy-in to 
the survey effort from various levels within the university. The Advisory Committee for 
the Faculty Survey was explicitly established to help the survey effort in that each 
member could be a spokesperson within his or her department or school to other 
administrators and faculty. In addition, the establishment of a Committee would help get 
the attention of the top decision-makers. The Advisory Committee was also helpful in 
reviewing wording that needed to be precise and appropriate for the respondent 
community (in particular, acronyms and building, place, and office nicknames). 

The HRES Faculty survey project ultimately included four teams: 



Table 4 

2003 Harvard Faculty Housing and Transportation Survey Team 


1. Core Team 


HRES Staff; Office of Budgets, Financial Planning, and 
Institutional Research staff 


2. Outside 

Consultant Team 


Survey research specialist; architectural consultant; Information 
Technology consultant 


3. Internal Advisors 


Faculty members with area expertise or methodological expertise; 
Office of Instructional Research and Evaluation 


4. Advisory 
Committee 


Deans or Directors representing each of Harvard’s 10 schools 



General Roles and Responsibilities of Survey Team Members 
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For any survey project, there are a number of survey functions that must be covered, and 
it would not be uncommon for survey team members to have overlapping duties. With 
paper surveys, an institutional researcher might do all the work alone ‘from A to Z,’ 
including pre-survey tasks such as copying the survey instrument, stuffing and stamping, 
and walking it to the post office. By contrast, web surveys involve tasks that require 
specialized skills and it is more likely that there would be at least two persons involved — 
a survey research specialist and an information technology specialist. 



Table 5 

Roles and Responsibilities of Survey Team Members 


1. Executive sponsor 


Senior administrator (President/Provost, VP, Dean), often the 
person who signs the cover letters to participate in the survey 


2. Project director 


Head of the department/office charged with administering the 
survey 


3. Project manager 


Person with overall responsibility for the survey effort 


4. Content specialists 


Identify themes and topical areas, determine questions 
(see below) 


5. Survey research 
expert 


Overall knowledge of survey research, with special 
responsibility for appropriate question wording, survey design, 
and web design. May be used to consult on all aspects of 
project including selection of appropriate survey population. 


6. IT specialists 


Survey layout, HTML programming, manage other IT 
functions, e.g. host survey, create final database 


7. Keeper of email 
list 


Knowledge of Human Resources or Registrar’s databases, 
assembles email list of survey population. 


8. Communications 


Manage all communications between sponsoring administrator 
and target populations (e.g., launch letter, reminder letters, 
closing letters). 


9. (Sales Manager) 


(Note that some consulting groups work through a sales 
manager who oversees many details but does not do the actual 
work.) 
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1.3. Defining survey content. (Led by Content Specialist and/or Survey Researcher.) 

If the project involves an original survey, a large portion of the survey effort should be 
devoted to defining the survey content. Issues of policy-level importance to the 
university will likely inform the content. It is important to focus on the questions that 
your audiences want answers to; questions should yield results that are useful, not simply 
interesting. In addition, if you have multiple audiences for your survey, each with their 
own agenda, take care to combine only related topics into a single survey at the risk of 
frustrating your respondent population. 

Here are some of our tips for defining survey content: 



Table 6 

Tips on Defining Survey Content 

1. Identify overarching themes that are of importance to executive sponsor and other 

internal decision-makers. Seek early and timely confirmation/approval from survey 
sponsor and other decision-makers. 

2. Use focus groups or interviews with members of target population to confirm themes 

and test sensitivities, if necessary. 

3. Write questions using the assistance of topical experts to be sure that questions will 

yield meaningful results that will inform policy setting, changes to business practice, 
etc. Avoid jargon, shorthand, and technical terms unless you are certain this language 
will make sense to everyone. 

4. “Paper test” questions with members of target audience to be sure that they (a) make 

sense to representatives of target audience and will be answered in the intended 
manner; (b) do not offend, if the topic is sensitive; (c) wording is appropriate for the 
times and the culture (e.g., ‘Black’ vs. ‘African-American’); (d) fixed-response 
questions include all the relevant options; (e) survey time does not exceed your goal 
length. (See Section III. 10.) 

5. Revise questions as appropriate. Whittle down questions to meet goals related to 

survey length. 

6. Provide questions (with parenthetical instructions about type of question, skip 

patterns, layout, etc.) to survey designer for programming. 

7. Prepare the survey designer and the web programmer from the very beginning that 

the survey might need a last-minute sign-off from a high-level administrator. 
Consequently, last minute changes may be involved, both big and small. 
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1.4. Choosing Web Surveys over Paper Surveys 

Although this paper assumes that you have already made the decision to use a web survey 
format, it is helpful to be able to defend this decision. At some universities, students, 
staff and faculty have very recently come to expect campus communication to take place 
over the web, and survey respondents appreciate the convenience of web surveys. 
However, there are certain populations and scenarios where web surveys might not be the 
best option. 



Table 7 

Advantages and Disadvantages of Web Surveys 


Advantages 


1. Savings in printing, postage, data entry. 


2. No data entry errors from hand-entry. (However, poor programming could lead to 
lost data.) 


3. Shortened timeframe to administer surveys (3 weeks with web surveys, vs. 6 weeks 
or more with paper surveys). 


4. Easier and cleaner to provide skip patterns or survey sections customized to different 
respondent populations. 


5. Almost immediate access to data for analysis. 


6. Can easily link to background data, if appropriate (e.g., gender, grades, rank). 


Disadvantages 


7. Need programming and IT expertise. 


8. Certain populations are not comfortable with using personal computers (senior 
faculty). (Note: at many campuses, this is becoming less of a problem.) 


9. Must have accurate email lists (e.g., a problem with students who use hotmail or aol 
accounts instead of school emails, with parents, with alumni). (Note: this is 
becoming less of a problem with many university populations, but could still be a 
problem with some, for example, part-time students.) 


10. Web surveys are not recommended for email software that doesn’t support web 
access. Must be able to click on a .url provided in an email and to have it bring 
respondent to a web page. PINE is an example of an unacceptable email software. 


11. There may be problems finding software that is appropriate for both PCs and Macs, 
or developing surveys that run on both platforms. 


12. Data provided via a web survey are not anonymous , although the survey 

administrators may choose to keep the results confidential. (See Section III. 11.) 
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1.5. Selection of Survey Population 

As anyone who has ever done IPEDS reporting, created a factbook, or responded to a 
data request on population numbers knows, there is more than one way to define the 
populations of students, faculty, alumni, parents, and staff. It is, of course, important to 
get the survey to the correct people, but it is also important to be able to document and, 
possibly, replicate the population selection in order to calculate the survey response rate. 
Often, an HR or IT person who is unfamiliar with the goals of the survey is the one to 
work with the administrative database to select the population and create the email list. 
Build in a way to double-check the population selection; for example, you might 
summarize the background data of the population and compare to a factbook table, or 
search the database to make sure that people who fall into a specific category are 
excluded. Here are five pitfalls that we have encountered: 



Table 8 

Pitfalls of Defining Populations 

1. Be careful to exclude the non-resident population if inappropriate for survey goals 
(e.g., overseas students, faculty on sabbatical). Example: if you are designing 
questions that get at satisfaction with or use of local resources, stick to the ‘on 
campus’ population. Note that a university email addresses can mask the fact that 
the recipient is at a remote location, so it might be helpful to screen by mailing 
address. 

2. For a survey of parents, you must decide if you will survey divorced parents 
separately, or only one parent per student. 

3. Will the list of respondents change during the lifespan of the survey (e.g., any 
student enrolled in a class throughout the term, vs. all students enrolled in a class 
as of the opening day of class), or even from the time you select the population to 
the beginning of the survey? 

4. Select on the proper variable to define your population. In one project with which 
we are familiar, a University variable that was thought to distinguish PhDs from 
other degree- seeking students was used, only to learn that this variable was not 
valid for one of the University schools. Thus, in that school, MA students were 
inadvertently surveyed along with PhD students. 

5. It is important to provide a background dataset with ids and demographic 
information on all recipients for response-rate analysis, analysis of 
representativeness of the respondent population, and any merging of survey 
responses with background data. Keep this file for reference. 
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II. Designing a Web Survey 

II. 6. Questionnaire Design 

The presiding rule of questionnaire design is to make it simple for the respondent to fill 
out. The questionnaire should be ‘eye candy’ - nice to look at, clear, flowing, easy. No 
incentive is high enough to induce respondents to waste their time on a confusing and 
frustrating questionnaire! There has been a lot written over the years on tips for paper 
questionnaire design, and many of these ‘rules’ still apply for a web questionnaire. 
However, designing a questionnaire for the web also has some additional and unique 
challenges. For one thing, everything is bigger and closer on the web. Things that are 
out of line look particularly jarring, and font and color clashes are right there to confront 
the respondent. 

The famous information design specialist Edward Tufte points out that, contrary to what 
you might think, a much smaller amount of information can be displayed on a web screen 
compared to a piece of paper. Keep in mind that because of all the headers and footers 
provided by Windows applications, sometimes as little as one-third of the screen might 
have your questionnaire on it. 
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Table 9 

Tips on Web Questionnaire Design 

1. Keep survey length to something that can be done in 25 minutes or less. This is the 
ideal, though it is not always possible. 

2. Manage the respondents’ expectations of survey length by telling them where they are 
in the survey, e.g., ‘Page 1 of 5,’ ‘Page 5 of 5.’ 

3. No matter what the length, some respondents will not complete the survey. This 
appears to be a problem of greater magnitude than previously for paper surveys. Be 
careful of the order of your sections, and do not include crucial questions in the final 
sections of the surveys. (Recommendation: put crucial demographic questions in the 
middle of the survey.) 

4. The less typing for the respondent, the better. There is some evidence that respondents 
tend to skip drop-down questions and write-ins for short answers (e.g., age, gender). 

5. Although we prefer to display all options on the screen and not use drop-down boxes, 
there may be situations where you prefer to use a drop-down box for a very long list of 
options. (For example, towns, states, occupations.) If you use a drop-down box, be 
sure that the default which appears on the screen is something like ‘Click Here’ and not 
a valid survey response such as ‘Boston,’ or ‘Strongly Disagree.’ You don’t want the 
respondent’s choice to be unnecessarily influenced by what they see on the screen. In 
addition, careless programming could turn a non-response into the default response. 

6. Also avoid write-ins for short answers if you can provide a coding scheme for the 
respondent. Respondents will rarely type their short answers the same way, which will 
involve a lot of hand re-coding at the analysis end. 

Example: Gender , vs. ( better ) Gender O Male O Female. 

7. Use a delicate hand with color and only to help your respondent navigate the survey. 
For example, every other line in a long grid of questions can be shaded, and section 
headers or header and footers can be in a different color than the questions. Be aware 
that unusual colors might not appear the same to all end-users because of 1) the color 
palate of the end-user’s computer, 2) possible color-blindness. 

8. Restrain the inclination to use images. They distract from the most important part of 
the questionnaire — the text. In addition, a file with images might load slowly. 

9. At least one general comment question at the end of the survey is useful — it provides 
good quotes to accompany the survey data analysis. In addition, it makes respondents 
feel that their opinions are valued. 

10. Each page should have a small distinctive logo/header on the top or bottom. A ‘help’ 
email or phone number should also be prominently displayed. Be careful, however, 
that the header/footer does not take up too much of the viewing page. 
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II.7. Choosing HTML Software and Format 

We have been involved in web projects that rely on many kinds of software to 
program the HTML: Cold Fusion, Dreamweaver, SurveySolutions, SurveyMonkey, and 
SurveySelect. None are perfect, but most can be adequately customized if the 
programmer is willing to take the time. Now is the time to start thinking about the fact 
that a programmer is generally just a programmer, not a survey research and 
questionnaire design specialist or a graphic designer. No software will get you what you 
need if the survey researcher doesn’t take the time to work closely with the programmer. 
More will be discussed on this in the next section. In addition, don’t assume that the 
programmer and survey researcher speak the same language with regards to coding, etc. 





Table 10 

Thoughts About Software 


1 . 


Own programming vs. canned packages. Each canned package has some sort of limitation - 
some that will jeopardize the integrity of the responses, and some that will affect the graphic 
look of the design. 

Example 1) Once a question is answered, respondent can’t erase responses to question 
entirely, although can change responses. To avoid this, provide a ‘not applicable’ or some 
such response so that respondents can decide NOT to respond at all after initially giving an 
answer 

Example 2) Might be initially difficult to evenly space a grid horizontally, or control 
hanging indents. 


2. 


Test the time it takes for each survey page to load (especially if using images); keep it short. 
Test using both a high-speed line and a dial-up line. 


3. 


Check to be sure survey works on both Macs and PCs. 


4 . 


Determine which browsers are optimal, including which version of the browser, and give 
this info to respondent in the cover page (i.e., Netscape 7.0, Internet Explorer 6.0). Provide 
respondent with the link to download current software, if necessary. 
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5. Output of data: 

■ It must be possible to export data to Access, an SPSS datafile or other statistical 

software, a comma or semi-colon delimited data, or a CSV file. 

■ Excel is not recommended due to size limitations for both the dataset and for individual 
text cells, but Excel works fine if there are a small number of respondents, the survey 
is short in length, and there are no lengthy comment fields. 

■ Comments (longer essay questions) work well in Access (using memo fields), or Word, 
and should be associated with an ID field in order to link with background and other 
survey data. 



II. 8. Technical Issues of Turning a Survey Questionnaire into a Web Survey 

In our experience, even giving the programmer a paper questionnaire that is fully 
designed and to scale is not enough. If you are doing the programming yourself, start by 
reviewing the basics of questionnaire design. (Dillman’s book Mail and Internet 
Surveys: The Tailored Design Method , John Wiley, 2000, is still the ‘bible.’) Then seek 
any help you might need on graphic design for the web from a Webmaster or someone in 
the publications department. Think about how you will analyze the questions at the end 
to help you program the skip patterns correctly. 

In the previous section, we mentioned that the programmer and survey researcher 
don’t necessarily share an understanding of data technicalities. After all, the 
programmer’s job ends when the survey administration is completed which is when the 
analyst’s job begins. In one project of which we are aware, the survey researcher 
indicated that dichotomous variables were to be coded as l=yes, and 2=no (hoping to 
avoid a situation where the programmer used l=yes and no is assigned to missing). The 
programmer used dichotomous codes, but relied on a programmer’s default, l=yes and 
0=no (and didn’t change the codebook). This problem could have been avoided if the 
researcher had tried a little harder to think like a programmer as, in our experience, it is 
more difficult for a programmer to think like a data analyst. 

In another project, a skip pattern was followed by a ‘mark all that apply’ type 
variable. If an answer wasn’t marked, the value was assigned as missing. The 
programmer did not understand that respondents who skipped must not be included with 
those who did not skip but who (actively) chose not to mark this option. 
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Table 11 

Turning a Survey into a Web Product 

1. You must decide if the respondents will be required to respond to any or all of the 
questions. Requiring anyone to respond to the survey as a whole as well as 
individual questions often does not fit with the University’s ethos. It also might 
conflict with instructions you receive from your Human Subjects Review 
Committee. (See Section III. 11.) In addition, with certain populations, it could 
simply annoy the respondents. Use selectively, if at all. 

2. You must decide if respondents will be able to exit and re-enter the survey, and if 
yes, whether or not they will be able to change any or all of their responses. If the 
survey is created as multiple pages, as described above, one method would be to 
allow the respondents to re-enter the survey but only be allowed to see and change 
their responses going forward (and not the pages already submitted). Be sure the 
respondents know whether they can exit and re-enter. 

3. Pre-determine the variability of the target population’s technical knowledge about 
the survey subject. Do you want or need to provide a pop-up glossary of terms? 

4. Select a graphic design template that is emblematic of survey sponsor’s image or 
the image of the college or university. 

■ Choose distinctive color scheme and appropriate fonts for headers and question 
text. Get help from a professional graphic designer, if necessary. 

■ Consider shading alternate rows in big grids of questions. 

■ Include good size margins to make page readable. 

5. Take time to be sure the programmer understands the questionnaire in order to 
program skip patterns correctly. 

6. Create the survey as multiple pages, and not as one long page. This helps the 
respondent navigate through the survey more easily. In addition, each individual 
page can be submitted (and captured) when completed, which will prevent the 
loss of data if the respondent doesn’t finish the entire survey if the system crashes. 

7. Survey designer should provide codebook or data dictionary of survey questions 
and possible responses for programmer to work with. All ‘click’ responses 
should be stored as numeric even when response choices appear as text to the 
respondents. Write-ins will be text. 

8. Obtain a print copy of the final web survey for inclusion in a survey report and for 
historical record. Note that some software packages will not provide a pretty print 
copy - what looks fine on the web may not print well on hard copy. It may be 
especially important to get a paper copy if the survey resides on a server that is 
not under the control of the survey group (e.g., another university office, an 
outside consultant). 
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III. Administering the Web Survey 

III.9. Technical Issues of Administering the Web Survey 

There are many small decisions to be made about how to administer the survey. 
Again, don’t rely on your programmer’s judgment, even if they are experienced at web 
surveys, because many of the decisions must be tailored to the culture and style of your 
University. 



Table 12 

Administering the Web Survey 

1. What is the desired timeline for the survey? We have found that 3 weeks is about 
right, with emails sent 3 to 4 days apart. We adjust the timeline based upon the 
response rate. If the response is flagging, we might send a reminder email one 
day earlier than originally scheduled. 

2. How many contacts? We have been contacting the respondent 4 times: a pre- 
letter from a top administrator (President/Provost, VP, Dean), a launch letter with 
the .url for the survey, a follow-up email only to those who have not responded, 
which also announces the end of the survey a few days hence, and a thank-you 
letter that goes to all recipients. 

3. Detail on suggested email contacts 

■ Optional pre-survey notification letter from top administrator to notify the 
population that the survey is coming and why it is important to the university. 

■ Launch letter/email containing the survey link and contact email in the event 
that any recipient has questions/comments. 

■ Pre-assigned unique URLs allow survey team to send 1 or 2 follow-ups to 
only those who have not responded. Sending reminder letters to those who 
have completed the survey will only cause frustration, confusion, and a slew 
of emails to project manager. The logon process can be used to select those 
who need reminders. 

■ 2nd follow-up could also be a notification of closing letter; “The survey will 
be closing on May 6 th at 5.00 p.m.” 

■ Thank you email goes to all recipients. Include information on response rate 
or something learned from the survey, if desired. 

■ Place “survey is closed” page at the survey address for latecomers to see. 



4. Create one survey alias address, for example, ‘Facultysurvey@harvard.edu.’ 
Typically, alias addresses are used for all communications from university 
officers who don’t want returned email coming back to them. Be warned that the 
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authenticity of the email could be in question if an email letter from a top 
administrator doesn’t have a matching return address (in the ‘Reply-to’ area). 

5. Choose an appropriate subject line for all email communications. We have used: 

■ A message from Provost Smith 

■ A reminder from Provost Smith 

Some subject lines with valid meanings may be identified by certain browser’s as 
junk mail, e.g., ‘congratulations,’ or ‘confidential.’ 

6. How will respondents enter into the survey? We have used a login via PIN 
authentication, and unique URLs for each respondent. 

7. Proper programming is needed to ensure that one person submits only one survey. 
Once in and done with a particular ‘page’ of the survey, respondent can’t enter 
that page again, avoiding multiple submissions. 

8. What constitutes a completed survey? 

■ Suggestion: after completing each page, have a click button that submits this 
page. (Could have 2 buttons, ‘Save and logout’ and ‘Continue to next page’) 

If re-enter survey, enter at beginning of first uncompleted page (entry on that 
page not saved). 

■ Suggestion: all respondents who have not submitted the final page of the 
survey should receive reminders. 

9. How are respondents required to navigate the survey? We prefer something in 
between requiring respondents to answer questions in a fixed order or answering 
questions at random. We use the fixed order of pages method - respondents can 
jump around on a particular page, but must do page 1 and submit it before going 
on to page 2. 

10. Time of day to send emails: 

■ Evening often works well when respondents are done with the business of the 
day and ready to ‘relax’ and look at non-work emails. 

■ If IT folks send the emails, it will depend on their schedules and the 
acceptable number of emails they can send at one time. The server must have 
the resources to temporarily store all of the emails until the respondents 
download them, and to do both the task of sending out these emails and other 
server chores. If not, it may be necessary to send out the survey in waves. 

11. Day of week to send emails: Differs from survey wisdom learned from paper 
surveys. It is acceptable to have web surveys arrive at any day of the week, 
including weekends. 
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12. Response Rate report while the survey is live. Use the time-date stamp that 
comes with each response to analyze the data, for example, responses by School 
and Faculty Rank. Used to: 

■ help target dates for reminders 

■ change or customize the reminder text 

■ catalog for future projects the days and times that your survey population 
prefers to respond to surveys 

■ manage the expectations of top administrators as to the pace of the survey and 
the eventual response rate. 



III.10. Pre-Testing 

Pre-testing takes time, but it is a worthwhile step. It’s akin to always asking for 
references — even if you have never had any problems, you know that it’s inevitable that 
you will someday if you don’t ask for references. Similarly, you should always have 
someone from the population you are surveying look at the questionnaire with fresh eyes. 



Table 13 

Pre-Testing 

1. Paper testing. When survey is still in the paper-stage, ask individuals or assemble 
a group of respondents to look for terms that they do not understand, confusing 
question wording, and obvious questions omitted from the survey. Be attentive to 
their comments, but remember that they are only a small group of people; your 
overall sense of the survey structure should predominate. 

2. Before the web survey is finalized, have individuals take the survey using a 
variety of different computer systems, e.g., Macs, PCs, dial-up, high-speed 
connection, and with different browsers. All should test whether the survey flows 
properly, whether there are any problems with going from page to page, pop-up 
glossary boxes, or incorrect skip patterns. 

3. Remind web programmer regularly that there will be some last minute changes 
based on feedback from pre-testing. 
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III.ll. Privacy Issues 

There are two sides to the privacy issues. The biggest issue of privacy is vis a vis the 
respondents. Every school has a Human Subjects Review Committee/Institutional 
Review Board (IRB) that is set up to review all types of projects involving human 
subjects. Human Subjects Review is very prominent for projects and experiments 
conducted by a Medical School or department of Psychology. In our experience, at many 
schools, administrative surveys are often exempt from Human Subjects Review. It is 
important to determine the position of your specific IRB. We found the 
recommendations of the Harvard IRB to be very helpful for determining the appropriate 
language to be used in email communications. 

The second issue of privacy is for any consultants that you might use. It is important 
to have a clear agreement regarding the capture, storage, and destruction of the survey 
data with any survey research or web programming consultants. Some schools are also 
requiring survey research consultants to have Human Subjects Review Certification. 

(This can be done over the web at National Institute of Health’s website.) In addition, it 
is important to discuss copyright issues with the consultant. A web-programming 
consulting firm might typically put a copyright on their survey product although your 
university would like to retain the copyright; be sure to clarify their company policies. 

The arrangement with a survey design consultant is different. Most survey questions 
that we are aware of are not copyrighted (e.g., CIRP questions). The unwritten 
convention has been that survey questions fall into the common domain, and survey 
researchers are even encouraged to reuse them. However, university protocol might 
require a web product to have a copyright notice on the bottom. This can be a problem 
because a survey research consultant legally owns the copyright to any questions that he 
or she created. One solution might be for the survey research consultant to have an 
explicit agreement to retain the copyright to the questions he or she designed, but to give 
the right to the university for the survey instrument as a whole. 
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Table 14 

Privacy Issues: A Harvard Example * 

1. Survey was deemed exempt by the Harvard Institutional Review Board 
because the survey did not meet the Federal definition of research. The 
survey was not ‘a systematic investigation designed to contribute to 
"generalizable" knowledge’ as the results were being used for in-house 
planning. 

2. The Harvard attorneys thought that it was unnecessary to include consent 
language for the recipients; login was considered consent. 

3. They also recommended including language to assure respondents that their 
responses will not be personally identified with them other than to the survey 
research analyst, (i.e., anonymous). 

4. Results will be aggregated for the University report - no individual data will 
ever be released, (i.e., confidential). 



5. Any comments made by respondents that include either the name of the 
respondent or the names of university personnel should have the names 
removed before being circulated. 



* Seek specific advice from your university attorneys or IRB. 
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III. 12. Survey Incentives 

With a paper survey, the administrator can include an actual object in the mailing. 
Fairly recently, in the commercial world, the ‘going rate’ was still $1 or $5. Diliman 
always stressed that pre-payment was more powerful than post-payment (after 
completing the survey). However, we are not aware of pre-payment ever having been a 
norm for university surveys. 

A number of schools that we are familiar with have gotten good boosts in survey 
response from offering creative and desirable gift incentives. A clever way to combine 
an incentive to complete the survey with an incentive to do so earlier rather than later is 
to have those who complete the survey early entered multiple times in a lottery. The 
introduction could say something like: 

Complete the survey by Tuesday and be entered into the gift lottery 10 times! 

Complete the survey by Thursday and be entered into the gift lottery 5 times!, etc. 

This technique has been used with student surveys and alumni surveys. It does not 
seem particularly appropriate for a survey of the faculty or parent population. Let the 
culture of your school be your guide as to when incentives are and are not appropriate. 



Table 15 

Incentives 

1. Be mindful of IRS rules on gifts and imputed income. At Harvard, no cash is 
allowed. 

2. Small gifts should be considered a ‘thank-you,’ but not truly an incentive 
(e.g., a coupon for free coffee and muffin). 

3. Use a lottery to draw prizes. 

4. Two suggested types of gifts: 

■ Offer a significant number of opportunities to win good-value gifts (e.g., 40 
gifts valued at $20-$40 each, like a portable CD player). 

■ Offer one large grand prize such as a travel certificate, a few moderate-priced 
gifts (university chairs are nice for parents or alumni), and a market basket of 
very small gifts. 
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5. Mail gifts to recipients. Requiring the gifts to be picked up can take a surprisingly 
large amount of coordination and staff time. 



III.13. Communication and Technical Support 

Inevitably, something will go wrong in the administration of the survey. Hopefully, it 
is something technical that you can control and correct. 



Table 16 

Communication and Technical Support 

1. Here is a short laundry list of things that could go wrong: 

■ the links don’t work 

■ the recipient does not know how to download needed software 

■ there are server problems when sending out too many bulk emails at once; 
(possible solution: send emails out in waves) 

■ the server could crash while the respondent is logged in 

■ the respondent’s computer doesn’t have enough memory to load survey 

■ the survey looks out of alignment on respondent’s computer. 

2. The survey instrument should prominently display the phone number and/or email 
address of a technical contact person for respondents who encounter problems 
with web administration. Respond promptly to every inquiry. Rule of thumb: 
respond to every query within 1 business day, if not sooner. The technical 
support person should document the problems and summarize for the project 
manager. 

3. Some respondents will write personal notes about the survey content to the 
technical contact. These comments should be forwarded to the project manager 
to be read. 

4. Surveys sent to incorrect or outdated emails will be returned. The technical 
support person should catalog this since it affects the calculation of the response 
rate. 
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III. 13. Output of survey data 

If the programming was done correctly (skips and assignment of missing values), this 
step is almost a non-step. One of the great advantages of web surveys is immediate 
access to a final (or even interim) dataset. In Section II. 8. we talked about the importance 
of the survey researcher and programmer speaking the same language. Here’s another 
example: be sure to explicitly tell the programmer that you want data values output as 
numeric and not as text (i.e., 1-2-3, and not ‘Always - Sometimes - Never). 

IV. Reportage 

Don’t let anyone tell you that all they need are the frequencies and that no report is 
necessary! The frequencies and means are still just data, and not yet useful information. 

Table 17 

Questions About Survey Analysis, and Reportage 

1. Who will be doing the analysis? 

2. Create a full set of frequencies and crosstabs as a paper appendix for posterity’s sake. 
Paper is still more portable than a datafile between offices and over time. 

3. A dataset is like a fingerprint — no two analysts will label, recode, and refine the 
dataset in the same way. Keep (and share!) documentation, e.g., calculating a mean 
of hours per week, where the top category, 20 +, is recoded as 22/hours per week. 

4. A decision should be made as to whether other University personnel will have access 
to the dataset to pursue analyses related to the project, or even unrelated goals. If you 
have developed recodes or created new variables that are crucial to the analysis, be 
sure to have them documented so that your analyses aren’t questioned when others 
cannot duplicate them. 

5. It is true: a picture is worth 1,000 words. Take the time to become an expert designer 
of charts and tables. 

6. All of your survey results will appeal to your direct ‘clients,’ those who initiated the 
project. Small parts of the survey will appeal to other University offices that may or 
may not have been one of the original intended audiences. Be creative about who 
might be interested in which pieces of data. 

7. Some of the audiences for the Harvard Faculty Survey were: 

■ Harvard Real Estate Services 

■ President’s Office 

■ Vice Presidents of Administration and Finance 

■ Office of the Provost 

■ Deans of Faculty Affairs (each school) 

■ Harvard Mortgage and Educational Foan Office 

■ Harvard Transportation and Commuter Services 
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